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ABSTRACT 

System  Concept 

A  system  has  been  developed  whereby  active  LADAR  and  passive  imaging  data  are 
registered  in  hardware  at  the  pixel  level.  For  the  sake  of  discussion,  the  sensor  is  herein 
called  a  “LADAR/EO  pixel-level  Fusion  Sensor”  or  LEFS.  This  sensor  produces 
structural  and  spectral  data  embodied  in  one  dataset,  permitting  composite  feature 
extraction  and  visualization. 

The  combined  use  of  LADAR  and  passive  EO/IR  data  has  well  known  synergism.  Our 
system  fuses  these  two  data  types  in  a  novel  way  such  that  real-time  performance  is 
possible.  This  significantly  enhances  the  ability  to  quickly  and  correctly  identify  targets. 

By  fusing  the  range  data  from  the  active  sensor  with  the  pixel-registered  imagery, 
registered  3D  images  are  available  in  real  time  (Figure  1).  Each  pixel  is  coded  with  full 
spectral  information  combined  with  structural  information  obtained  by  the  active  system 
(Figure  2).  The  resulting  fully  aligned  high-dimension  feature  vector  enhances  target 
recognition  and  permits  dense  point  matching  for  precise  image  mosaicking. 

A  significant  benefit  is  in  combining  the  ability  of  pencil  beam  active  systems  to  work  at 
long  ranges  and  to  penetrate  obscurants  with  the  passive  array’s  wide  instantaneous  field 
of  view  at  increased  resolution.  One  application  in  which  this  has  had  enormous  benefit 
is  in  observation  through  partial  or  intermittent  obscuration;  e.g.  with  partial  cloud  cover 
or  foliage.  Reflected  radiation  associated  with  features  within  gaps  in  the  obscuration  are 
sensed  passively  while  at  the  same  time  the  active  pencil  beam  efficiently  maps  structure 
within  the  revealed  region  (Figure  3).  Pixel-level  registration  of  data  permits  extended 
regions  to  be  mapped  by  combining  temporally  or  spatially  diverse  collections  (Figure  4). 

Applications 

Military  application  for  which  this  technology  is  being  developed  or  assessed  include 
precision  tactical  targeting,  Precision  Controlled  Reference  Image  Base  (CRIB) 
production  and  Automatic  Registration  of  targeting  data  into  the  CRIB. 

Civil  applications  include  3D  city  modeling,  real-time  airborne  mapping,  post-disaster 
reconnaissance,  floodplain  and  coastline  mapping,  drug  interdiction  target  detection, 
environmental  monitoring,  and  search  and  rescue. 


Paper  presented  at  the  RTO  1ST  Workshop  on  Massive  Military  Data  Fusion  and  Visualisation:  Users  Talk 
with  Developers ”,  held  in  Halden,  Norway,  10-13  September  2002,  and  published  in  RTO-MP-105. 
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Visualization  Implications 

Such  detailed  three-dimensional  multi-spectral  data  present  challenges  and  opportunities 
for  visualization. 

The  multispectral  character  of  the  data  presents  the  familiar  problem  of  presenting  high 
dimensional  data.  Adaptive  use  of  false  colors  is  the  most  obvious  approach.  In  the  case 
of  an  LEFS  system,  the  false  color  scene  is  wrapped  into  a  3D  structure. 

The  opportunity  to  recover  range-tagged  feature  data  through  partial  or  intermittent 
obscuration  permits  partial  3D  scene  reconstruction.  The  use  of  multiple  viewpoints 
enables  the  filling  in  of  the  inevitable  residual  shadowing  associated  with  a  given 
viewpoint.  One  potential  visualization  technique  is  that  of  selective  layering.  Using  this 
technique,  the  analyst  is  able  to  scan  or  step  through  layers  of  the  data,  either  in  the  range 
dimension  or  in  a  dimension  contoured  to  modeled  terrain.  In  this  way,  for  example,  the 
analyst  can  explore  the  scene  at  or  near  ground  level  by  suppressing  the  intervening  cloud 
or  foliage  obscuration.  Alternatively,  he  can  isolate  aircraft  or  other  objects  from  an  earth 
background. 

A  very  high  degree  of  compensation  for  motion  in  the  sensor  platform  is  achievable 
because  the  precise  range  is  obtained  from  the  LADAR  at  the  same  time  the  fully 
registered  precise  angle  data  from  the  passive  optical  system  is  collected.  This  eliminates 
the  parallax  problems  encountered  by  passive  systems  alone.  Thus  passive  images  can  be 
automatically  co-registered  as  proved  by  a  prototype  LEFS  system  at  Utah  State 
University.  Figure  5  shows  an  example  of  an  automatic  image  mosaic  reconstructed  on- 
the-fly  using  the  wire-frame  surface  structure  derived  from  LADAR  shown  in  Figure  6. 
The  vantage  point  of  the  viewer  of  these  Figures  happens  to  be  above  and  to  the  left  of 
the  point  from  which  the  data  was  collected.  Such  virtual  positioning  is  possible  only 
because  of  the  3D  nature  of  the  database.  Shadows  in  the  figure  are  regions  that  were 
blocked  from  the  view  of  the  instruments,  where  no  data  could  be  collected.  The  linear 
seam  on  the  right  side  of  the  lower  image  indicates  the  boundary  between  two  adjacent 
digital  camera  shots.  Brightness  was  deliberately  increased  in  the  image  on  the  right  of 
the  seam  to  help  locate  the  seam.  Note  how  perfectly  the  two  images  are  knitted  together 
because  of  the  georegistration  of  the  data. 

Ultimately  a  full  tomographic  capability  could  be  provided.  Such  may  have  applications 
in  dense,  complex  three  dimensional  scenes  as  might  be  encountered  in  urban 
environments.  For  example,  it  should  be  possible  to  construct  detailed  imagery  or  video 
of  individual  vehicles  or  people  as  they  move  through  crowded  streets,  which  intermittent 
blockage  by  other  vehicles  or  people,  buildings,  etc. 
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RANGE  DATA 
FROM  MULTI¬ 
CHANNEL  LADAR 


PASSIVE  IR 

IMAGE  DATA  FUSED  RANGE  IMAGE 


Figure  1.  Hybrid  3D  Image  via  Pixel-Level  Registration 


Figure  2.  Pixel-Level  Spectral  and  Figure  3.  Observation  through  Partial 

Structural  Feature  Set  Obscuration 


Figure  4.  Large-scale  image  fusion 
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Figure  5.  3D  images  mosaicked  using  LAD AR  data  shown  in  Figure  6.  Note  the 
seam  defined  by  two  images  of  differing  brightness. 


Figure  6.  Wire  frame  constructed  from  LADAR  data 
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SYMPOSIA  DISCUSSION  -  PAPER  NO:  12 


Author’s  Name(s): 

Mr.  Alan  Steinberg,  Utah  State  University,  USA 

Robert  Pack,  Space  Dynamics  Laboratory/Utah  State  University,  USA 

Question: 

Range  data  and  image  data  were  precisely  rectified  in  this  application.  How  difficult  is  it  to  merge  data 
from  different  platforms? 

Author’s  Response: 

The  ability  to  co-register  data  sets  is  more  straightforward  if  you  have  the  same  sensor  types.  With 
different  sensors  it  is  difficult  to  have  such  a  precise  correlation. 

Comment: 

You  can  see  how  knowledge  evolves  over  time  by  observing  and  understanding  the  estimate  of  knowledge 
over  time. 

Comment: 

By  running  the  contour  map  of  sensor  performance  back  over  a  period  of  time  you  can  see  your  ability  to 
observe  or  be  observed. 
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Concept  of  Pixel-Level  Fusion 


•  Active  Sensor  -  LADAR 

•  Passive  Sensor  -  Electro-Optic  (EO)  Radiation 

•  Fused  into  One  Data  Set 


RANGE  DATA 
FROM  MULTF 
CHANNEL  LADAR 


PASSIVE  IR 

IMAGE  DATA  FUSED  RANGE  IMAGE 
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Enables  Observation  Through  Partial 

Obscuration 


12-3 


Employment  Concept 


Ground  Clutter 
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Fused  LADAR/EO  Image  1 

(Co-Registered  and  Georeferenced  at  the  Pixel  Level) 


LADAR  Image 


EO  Image 
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Fused  LADAR/EO  Image  2 

(Co-Registered  and  Georeferenced  at  the  Pixel  Level) 
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Orthoprojection  of  the  Two  3D  Images 
Combined  in  Geographic  Space 
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Profile  Views  in  3D  Viewer 


First  and  Second  LADAR/EO  Images  Combined 
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Visualization  Concept  - 
Data  Volume  Exploration  and  Clutter  Removal 
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3D  Images  of  Cluttered  Environment 


Vegetation  Canopy 

Terrain  (per  DTED 
or  detected  ground) 

Ground  Plane 
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Predict  Ground 


Bottom  of 
Canopy  Layer 
(Terrain  +  3m) 

Terrain  (per  DTED 
or  detected  ground) 

Ground  Plane 


Slice  Volume  at  Bottom  of  Canopy 


Bottom  of 
Canopy  Layer 
(Terrain  +  3m) 

Terrain  (per  DTED 
or  detected  ground) 

Ground  Plane 


Residual 
Shadowed 
Regions:  PD  =  0 
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Example  of  Canopy  Slicing 


Oblique  EO  Scene 


Orthoprojected  Using  LADAR- Trees  Removed 
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Two  Shots  Combined 


Trees  Removed 
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Target  Hidden  Under  Tree 
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Target  Exposed  After  Tree  Removal 
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Conclusions  (1): 

Pixel-Level  Fusion  of  Active/Passive  Data  for  Real-Timo 
Composite  Feature  Extraction  &  Visualization 


•  Pixel-Level  fusion  of  LADAR  +  EO  data 
facilitates  target  detection  &  recognition  in 
highly  occluded  environments 

•  Clutter 

-  tree  canopies,  clouds,  etc.  - 
can  be  removed  through  geometric  filters 
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Visualization 

Implication 


12-18 


Some  Visualization  Implications 


•  Target  Detection  &  Recognition 

>  Probabilities  of  Detection 

>  Probabilities  of  Target  Presence  (prior  & 
posterior) 

>  Exploitation  of  Negative  Data 

*  Temporal  Evolution  of  Estimation  and 
Expectations 

>  Out-of  Sequence  Data 

>  Situation  Projection 

>  Expected  Updates 
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Prior  &  Posterior  Detection  Probabilities; 
e.g.  Presentation  of  Negative  Data 


Target  Expectations 

Detectability  Map 

Map 


Prior  or  Posterior 
Probability  Density 

(e.g.  Track  Projection 
Uncertainty) 

p(x,t+n\t) 

X  =  Target  state  (type  & 
location,  etc.) 

t,  t+Tl  =  Update  times 

A  =  Action  plan  (sensor 
route,  sensor  controls, 
processing  controls,  etc.) 


p(x,  t+n  |  t)pD(x,  t+n\t,A ) 
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Some  Visualization  Implications 


•  Target  Detection  &  Recognition 

>  Probabilities  of  Detection 

>  Probabilities  of  Target  Presence  (prior  & 
posterior) 

>  Exploitation  of  Negative  Data 

•  Temporal  Evolution  of  Estimation  and 
Expectations 

>  Out-of  Sequence  Data 

>  Situation  Projection 

>  Expected  Updates 
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Latency  Issues: 
Out  of  Sequence  Data 


Initial  Estimates  of  Track  Histories  (MTI  only) 


Revised  Estimates  of  Track  Histories  (MTI  +  Imagery) 
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Adaptive  Information  Exploitation 

Process 


*  Acknowledge  uncertainties 

*  Estimate  length  of  time  for  which  decision 
can  be  deferred 

*  Estimate  probability  of  resolving  ambiguity  in 
time 

>  Given  current  collection  plan 

>  By  redirecting  current  resources 

>  By  adding  resource 

*  Estimate  Cost  &  Net  Utility  of  Candidate 
Action  Plans 

*  Select  &  Implement  Revised  Plan 
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Visualizing  Time-Varying  Situation 
Estimation  &  Expectation  (1  of  3) 


Iah- 

Current  Estimate  of  \ 

Present  Situation 

— "  )  /vS.  \\  ff  / 

Situation  Time  Estimated  History  Estimated  Phe^ 

pnt  Situation 

Projection  Situatfc 

»n 

Report  Time  (Plan  A)  Received  Reports  [ 

Expecte 

id  Coverage 

-40  -30  -20  -10  ( 

)  +10 

+20  +30 

+40 

Tgt  Selection  Decision  Tgt  Engagement  Decision 


12-24 


Visualizing  Time-Varying  Situation 
Estimation  &  Expectation  (2  of  3) 
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Visualizing  Time-Varying  Situation 
Estimation  &  Expectation  (3  of  3) 
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Conclusions 


•  Pixel-Level  fusion  of  LADAR  +  EO  data  facilitates 
target  detection  &  recognition  in  highly  occluded 
environments 

>  Clutter  -  tree  canopies,  clouds,  etc.  -  can  be  removed  over 
time  through  geometric  filters 

•  Enables  visualization  of  time-varying  situation 
estimation  &  expectation 

>  Presentation  of  Detection  Probabilities  (prior  &  posterior): 
e.g.  Presenting  Negative  Data 

>  Presentation  of  Temporal  Evolution  of  Estimates  and  of 
Expectations 

-  Out-of  Sequence  Data 

-  Situation  Projection 

-  Expected  Updates  &  Information  Evolution 
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Thank  You 
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